Research data supporting "Deep learning as optimal control problems"

We consider recent work of Haber and Ruthotto 2017 and Chang et al. 2018, where deep learning neural networks have been interpreted as discretisations of an optimal control problem subject to an ordinary differential equation constraint. We review the first order conditions for optimality, and the conditions ensuring optimality after discretization. This leads to a class of algorithms for solving the discrete optimal control problem which guarantee that the corresponding discrete necessary conditions for optimality are fulfilled. We discuss two different deep learning algorithms and make a preliminary analysis of the ability of the algorithms to generalise. We provide data and associate code for some of the examples reported in the paper.We acknowledge support from the Leverhulme Trust EarlyCareer Fellowship ECF-2016-611 ’Learning from mistakes: a supervised feedback-loop for imaging applications', the Leverhulme Trust project on Breaking the non-convexity barrier, the Philip Leverhulme Prize, the EPSRC grant No. EP/M00483X/1, the EPSRC Centre No. EP/N014588/1, the European Union Horizon 2020 research and innovation programmes under the MarieSkodowska-Curie grant agreement No. 777826 NoMADS and No. 691070 CHiPS,the Cantab Capital Institute for the Mathematics of Information and the Alan Turing Institute. We gratefully acknowledge the support of NVIDIA Corporation with the donation of a Quadro P6000 and a Titan Xp GPUs used for this research. We thank the SPIRIT project (no. 231632) under the Research Council of Norway FRIPRO funding scheme. We also would like to thank the Isaac Newton Institute for Mathematical Sciences, Cambridge, for support and hospitality during the programmes 'Variational methods and effective algorithms for imaging and vision' (2017) and 'Geometry, compatibility and structure preservation in computational differential equations' (2019) where work on this paper was undertaken. This work was supported by EPSRC grant no EP/K032208/1